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TOWARD  A  PROPER  LOGIC  FOR  PARALLEL  COMPUTATION 
IN  THE  PRESENCE  OF  NOISE 

Jack  D.  Cowan* 

Massachusetts  Institute  of  Technology 
Research  Laboratory  of  Electronics 

INTRODUCTION 

1  2  3 

The  work  of  J.  von  Neumann,  and  of  W.  S.  McCulloch,  M.  Blum, 

4 

and  L.  A.  M.  Verbeek,  on  the  construction  of  reliable  automata  from 
unreliable  components  has  demonstrated  the  effectiveness  of  introducing 
redundancy  into  the  structure  of  such  automata,  by  means  of  the 
■bundling"  technique.  That  is,  replacing  each  single-line  automaton  by 
a  parallel  net  of  single-line  automata,  subject  to  the  constraint  that 
“bundles"  in  the  multiple-line  automaton,  carry  the  same  amount  of 
information  as  do  individual  lines  in  the  single-line  automaton.  Thus  if 

XI. 

there  are  n  lines  per  bundle,  each  line  carries  l/n  of  a  bit,  com¬ 
pared  with  1  bit  per  line  for  the  single-line  automaton.  The  number  of 
lines  per  bundle  is  a  measure  of  the  redundancy  of  parallel  nets.  The 
informational  constraint  is  introduced  as  follows: 
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A 
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Figure  1 


Single-line  automata  are  replaced  by  multiple-line  automata  (Figure  1) 
that  possess  n  lines  per  bundle.  A  fiduciary  level  A  is  set  such  that 
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Laboratory,  Bedford,  Massachusetts. 


If  more  than  (l-A)n  lines  “fire®,  this  is  interpreted  as  "1*. 

If  less  than  An  lines  ®fire",  this  is  interpreted  as  ®0“. 

Any  intermediate  firing  level  is  interpreted  as  a  malfunction, 

"i». 

In  the  construction  used  by  McCulloch  et  al.  ,  the  same  coding  procedure 
is  followed,  but  the  midtiple-line  automaton  has  the  following  structure: 


Figure  2 


Thus  each  “neuron"  receives  inputs  from  all  lines  in  both  bundles.  We 
shall  not  enter  into  an  analysis  of  these  particular  systems  (see 

Lbfgren^  and  Cowan^)  but  merely  state  the  results.  For  both  construc¬ 
tions,  if  e  is  the  probability  of  malfunction  associated  with  each  com¬ 
ponent,  and  Pg  is  the  probability  of  malfunction  of  a  complete  net,  we 

obtain  the  following  relation: 
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Thus  P  is  a  monotonically  increasing  function  of  l/n,  and  in  both 
cases  lim  P  =  0.  But  l/n  is  the  number  of  bits  per  ■line",  and  is 

n-*00  y 

thus  a  measure  of  the  information  rate  (see  Shannon')  in  the  multiple¬ 
line  automaton.  So  these  results  both  have  the  property  that  the  rate 
is  zero  for  reliable  computation  in  the  presence  of  noise.  When  we 

consider  Information  Theory,  ^  this  result  is  very  surprising.  The 
noisy  coding  theorem  may  be  stated  as  follows: 

If,  for  a  communication  system,  there  exists  a  certain  maximum 
rate  for  transmission  of  information,  the  channel  capacity  C,  then 
for  transmission  rates  R  less  than  C  ,  it  is  possible  to  introduce 
redundancy,  independent  of  rate,  in  such  a  way  as  to  obtain  an  arbi¬ 
trarily  sm^l  error  probability,  P^.  At  rates  higher  than  C  »  Pg 

increases  with  R  -  C  . 
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0  C  I  R 

Figure  4 


The  results  of  von  Neumann  and  McCulloch  et  al.  would  seem  to 
imply  that  the  capacity  for  computation  of  information  is  zero.  These 

g 

results  motivated  Elias  to  attempt  an  information -theoretic  solution 
to  the  problem.  He  noted  that  only  the  odd-jot  functions  of  2-variable, 

3 

2-valued  logic  presented  any  problem  (see  Blum  ).  For  these  func¬ 
tions  a  block  code  (k  information  digits,  n  -  k  check  digits)  allows 
correction  of  all  errors  as  the  block  length  n  increases  indefinitely, 
only  when  the  rate  k/n  is,  at  most,  l/n.  (In  fact,  the  rate  can  never 
be  greater  than  l/3  for  correction  of  single  errors.)  Thus  block  coding 
is  no  better  than  the  iterative  coding  of  von  Neumann  and  McCulloch 
et  al.  On  the  basis  of  this  result,  Elias  hypothesized  that  the  computa- 

9 

tional  capacity  is  zero.  Petersen  carried  through  a  similar  argument 
for  binary  group  codes  and  achieved  essentially  the  same  results. 

We  shall  now  attempt  to  define  a  particular  information  measure 
relevant  to  computation,  as  distinct  from  transmission,  in  order  to 
gain  insight  into  the  nature  of  the  foregoing  results,  and  attempt  a 
solution  to  the  problem  of  realizing  a  nonzero  rate  for  reliable  computation. 


96 


COMPUTATIONAL  CAPACITY 


A  computing  device  differs  from  a  transmitting  device,  in  that,  in 
general,  the  former  is  ■dimension-reducing,"  while  the  latter  is 
■dimension-preserving."  Thus  we  can  characterize  computing  devices 
as  follows: 


Figure  5 

and  transmitting  devices  as  follows: 
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Figure  6 


97 


The  two  systems  can  be  distinguished  as  follows.  Consider  both  as 

mappings  from  an  input-space  X  into  an  output  -  space  Y  .  Then,  in  the 
absence  of  noise, 

(2.1)  For  transmission 


H  (X|Y)~oI/5IZI  a  0 

(2.2)  For  computation, 

H  (riY)  >  0 

The  quantity  HCSIT),  called  the  communication  entropy  of  ^ given  ^  . 
is  called  equivocation. 

We  can  also  define  the  mutual  information  rate  for  IT-'  i»' Y 

SCXil.3 

=  h(h;)- 
=  Htn-  H(riE) 


Clearly,  H  (  g  1 Y  )=  H  (T  |l)  implies  Hti;)=  HLZ) 

Since  H(;c)  =a#-Zlft{o3-eo^Rrt«3  is  in  a  certain  sense 

a  measure  of  dittAlL  ,  the  dimension  of  the  space  X(see  Renyi^^),we 

require  for  computation  to  occur  in  H  CJ)  <  H  {£)  . 

and  hence  Hix  12)  <•  .  That  is,  the  noisy  entropy  must 

be  less  than  the  equivocation.  In  general,  a  computing  sc.heme 
exists  when 


(2.3) 


HtZiT) 


in  the  absence  of  noise. 


(2.4) 


H  (XIX)  -c  HtXlT) 
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Consider,  now,  the  following  scheme: 


4  }  ) 


Figure  7 

or  equivalently  McCulloch's  chiastic  scheme. 


Figure  8 


This  represents  a  computing  scheme  in  which,  not  the  argument,  but  the 
function,  is  probable.  By  specification  of  the  we  represent  all  pos¬ 
sible  Boolian  logical  functions,  and  their  perturbations  by  noise  or  mal¬ 
function.  For  example,  suppose  (oj,  a^,  ay  a^)  =  (c,  e,  e,  1-e).  Then  we 

have  specified  the  following  scheme: 


Figure  9 


We  interpret  this  as  the  function  or  X2® 


Figure  10 
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together  with  the  fact  that  with  each  possible  input -output  pair,  there  is 
associated  a  precise  probability  of  error  C.  1 

We  can  formally  represent  this,  by  attaching  a  binary  symmetric 
channel  (BSC)  to  the  output  of  the  computing  device: 


Figure  1 1 

Of  course,  we  need  not  attach  a  BSC,  capacity  C^c  =  1  •* 

•ss  I  +•  6  I“6  )  to  the  output.  We  could  have 

used  any  noisy  binary  channel,  with  capacity  C  at  the  output.  For  sim¬ 
plicity,  we  shall  consider  here  only  the  former  case.  Furthermore,  we 
shall  consider  only  systems  with  two  inputs  {x^,  x^).  In  general,  of 

course,  we  can  have  any  number  of  inputs. 


a 

The  general  n  —  1  computational  scheme  (n  finite),  with  any  noisy  binary 
channel  at  the  output  is  being  considered  by  S.  Winograd,  Research  Lab¬ 
oratory  of  Electronics,  M.  I.  T. ,  and  the  writer. 
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We  are  thus  considering  the  following  scheme: 


Figure  12 

which  we  symbolize  as  ®  Y*  — |r-«  ►  ^  Since 

^  is  noiseless,  the  capacity  for  is  just  1  bit,  while 

the  capacity  for  Y  Z  is  ^85C  .  And  since  the  two  channels 

are  in  series,  the  maximum  rate  for  •  ^  is  just 

C  -Ij  ^(b«c  )  =•  C  C  •  That  is,  the  maximum  mutual  informa¬ 
tion  rate  that  can  be  obtained  from  the  noisy  computation  scheme 
-f-o ^  ^  is  just  the  capacity  of  the  dimension¬ 

preserving  channel  specifying  the  noise  in  the  scheme. 

We  can  compute  this  explicitly  as  follows:  we  note  that 

O  &  HCSi®Ki.l  <£  2. 

0  4  HIT  >  a 

H  1  T  I  Xi  ®  )  =  o 

s  h(s,®X2,1X)  s  h(Xi®s:2.) 

C  Xi®  X2.)  — '  H  ( Y-i®  SZi  ^ 

==.  HLY.)-  H(Y 


(2.5) 

(2.6) 

(2.7) 

(2.8) 
(2.9) 


0 

H 
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Prom  (2.7)  and  (2.9) 

H(Y;= 

That  is, 

^  =  Hm 

Clearly, 


I C  Si®  Sj-i  ^1=  ^  C^^S>^Z^■,T2 -  H  (Y liJ 

=  HCY)-H(YlZ) 


Thus  the  capacity  for 

=^m^CHCY)-HCXl-^3^  =* 


is  just  C."£i®'X2  ■; 

CbSC  •  (9-e.d.) 


From  this  simple  arialy.sis,  it  would  appear  that  essentially  the 
same  results  that  apply  to  the  noisy  computation  channel  apply  to  the 
noisy  transmission  channel,  namely:  up  to  a  certain  mutual  informa¬ 
tion  rate,  CfeSc>  is  possible  to  introduce  redundancy  into  the  sys¬ 


tem,  independent  of  rate,  and  achieve  an  arbitrarily  small  probability 
of  error  in  the  computation.  The  question  is,  of  course,  can  we,  in 
fact,  introduce  redundancy  into  a  computing  scheme,  so  as  to  realize 
this  nonzero  rate  for  reliable  computation,  and  if  so,  what  is  the  nature 
of  this  redundancy?  Before  trying  to  answer  this  question,  we  shall 

analyze  the  nature  of  the  measure 


THE  NATURE  OF  SCSii»Si',l,l 

For  transmission,  the  information  measure  that  we  have  used  is 
SZXjYJ  ,  which  we  interpret  as  a  measure  of  the  average  rate  at 
which  information  about  "points"  in^is  provided  by  reception  of 
"points"  iuX-  In  the  case  of  computation,  the  measure  that  we  have 
used  is  d  j  ^  ,  where 

^CXiSiX2j%3  “  HtX'i®Xl)-HCXi®Srj.l5,3 

=  Htll  -  H  £l.| 
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Furthermore, 

=  HtY)-H(YlX) 

h(z)-h^XI^3 

where 

T  = 

Thus 

H  ( I  1  X-,® Xt.)  =  H  ( I  U«  Xi®xO 

This  is  a  consequence  of  the  fact  that  H  C  I  Xi®Xi)=  O 

since  ^  is  noiseless.  Thus 

)-H(E|Si®Xz) 

=  hCI)-M(X|/.Zi®xO 

=  ^  Cif-XicXzil] 

=*#SC  f  if*J 

where 

rv*-' 

This  computation  measure  is  thus  a  measure  of  the  average  rate  at 
which  information  about  points  in  /o  Xi®  Xi  is  provided  by  reception 
of  “points'*  in  But  clearly,  points  in  ^0X1  ®3[.z 

correspond  to  point  sets  in  X']  ®  •  That  is  3  C.  j  ^ 
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is  a  measure  of  the  average  rate  at  which  information  about  point  sets  in 

2i®ri  ,  specified  by  ,  is  provided  by  reception  of  points  in  g  . 

Since  we  are  dealing  with  functions  that  are  only  probable?  this  is 

reasonable.  Given  a  X  £  Z  ,  we  wish  to  compute,  not  what  was  the 

most  likely  input  sequence  ,  but  what  was  the 

most  likely  input  sequence  subset,  or  equivalently,  what  was  the  most 

likely  .  Furthermore,  since  this  is  all  that  we  wish 

to  compute,  no  matter  where  malfunctions  are  located,  and,  for  example, 
in  formsd  neurons  they  can  occur  as  threshold  fluctuations,  synapse 

errors,  axon  errors,"*  as  far  as  6  C  ^  i  is  concerned,  these 

malfunctions  are  all  effective  at  the  output  of  y  . 

CODING  FOR  THE  COMPUTATION  CHANNEL 

We  now  return  to  the  question  of  coding  for  the  computation  channel, 
and  our  approach  is  based  on  certain  implications  of  the  noisy  coding 
theorem  for  transmission  channels.  The  whole  process  of  introducing 
redundancy,  encoding,  decoding,  and  so  forth,  can  be  represented  by 
the  following  set-theoretic  picture: 


encod  ir»o 


t  S  wa  V  S  S  i 

Figure  13 


Y 


We  note  in  this  context,  that  the  work  of  Shannon  and  Moore, ^  Kochen, 
14 

and  Allanson,  is  concerned  not  with  probabilistic  functions,  but  with 
functions  of  probabilistic  arguments,  and  is  different  in  character  from 
the  schemes  considered  here. 
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The  theorem  tolls  us  that  provided 

is  less  than  Cft&c  >  we  can  decode  with  arbitrarily  small  error  proba¬ 
bility  for  the  mapping  X"  ^  ^  ^ 

vided  <  Cftsc  .  Thus,  for  reliable  transmission 

,  or,  in  general,  Jfc^'ss  ,  where 

R=  SCXl'Zi  . 

For  computation,  R»  ,  and  by  analogy  with  transmis¬ 
sion,  we  might  expect  to  have  ML  =  J 

"message"  points  in  ;f'V)Xl®Xz.  =  i-e.: 


^o£t®Xa. 


f  oiz. 


enCoctiirt. 


«1 


«l£lditiO/l 

noiSti 


dcCod.\'r 


7 


Figure  14 

V/e  distinguish  betv/een  the  function  reconstructed  by  the  decoding 
operation  a^,  and  the  initial  function/.  Thus 
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The  function  ^  e  ®  Sla.  is  interpreted  to  be  a  mapping  of  2li®£2. 

into  a  space  containing  a  larger  number  of  elements  than  the  original 

function  space  jSi(S  •  The  subspace  (or  subset)  is  chosen 

as  a  "message"  subset.  Clearly,  we  are  dealing  with  redundancy  of 
function- rather  than  redundancy  of  argunnent.  Now  for  two-valued,  two- 
variable  logic,  £l«iXz.  is  a  space  with  two  elements,  and  hence, 

to  obtain  the  necessary  functional  redundancy,  we  require  a  space 

with  many  elements,  and  a  choice  of  only  of  these 

to  be  informationally  significant.  That  is,  we  have  to  construct  a  many- 
valued  logical  scheme  in  which  not  all  trutn  values  are  informationally 


significant.  This  requirement  is  consistent  with  our  location  of  the  noise, 
and  our  consequent  use  of  .  From  our  equation 

Z  =1  "S’  — .  it  is 

evident  that  Xq®  Xl  »  to  be  matched  to  the  noisy 

channel,  and  hence  redundancy  in  ^  is  required.  And  is  just  the 

function  space  of  .  It  must  be  realized,  however,  that  we  are  not  free 
to  perform  ^  "V  ,  introduce  redundancy  in 

then  perform  "g  2  .  On  the  contrary,  we  are  really  dealing 

with  XqiSXz.  ji'fl  ^  2  .  The  other  is  merely  a  conceptual 

device,  and  the  necessary  redundancy  may  only  be  introduced  into 
either  ,  or  y:  ,  or  both.  At  any  rate,  the  coding  scheme 

that  we  obtain,  in  general,  is 


noise. 


Figure  15 
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We  write  F  for  the  ad  hoc  function,  to  indicate  that  it  may  be  anything 

from  /  itself,  to  a  much  more  complex  function.  We  denote  by  JW 
the  encoded  message  set,  that  is  ss. 

tinguish  between  =  2^  8)  09  ,  where  Wl  sa  0^1  ®X'i  »  o  Xx  » 
which  we  call  coordinate  encoding,  and  jyi  :jfc  ryitS^JXi*  which  we  call 
pointwise  encoding.  We  assume  that  Otp  and  are  noiseless,  and  that 

*  g 

no  computation  results  from  The  necessary  conditions  to  be  satis¬ 

fied  by  the  scheme  can  then  be  stated  as  follows:  In  the  absence  of 
noise,  ,  and  is  CiJll  for  all  • 

For  coordinate  encoding  this  can  be  restated  as 

M*F  —  F  s  ^  if  I  and  M®F  is  [2  :  Q  for 

all  •Wj  - p  »  .  Furthermore,  in  the  presence  of  noise, 

CM)  ,  the  minimum  distance  between  points  of  satisfies  the 
Hamming  inequality,  namely  that  d  C  M  )  ^  2‘^+»  1  i  for  correc¬ 
tion  of  all'fi-uple  errors. 

MANY-VALUED  LOGICAL  SCHEMES 

Let  us  now  examine  various  forms  of  many-valued  logical  schemes. 
We  consider  initially  a  Post  logical  disjunction  x^  and  (see  Appendix) 

which,  for  the  3-valued  scheme  can  be  represented  by  the  following 
truth-value  matrix 


which  we  also  recognize  as  a  mapping  from  the  lattice  of 

Xn®X2— ••X  ,  where  X-  {1,1, -5]  : 
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Figure  16 

Let  us  choose  M=:  {1,33  ,  so  that  we  have  the  following 

coordinate  encoding  scheme  in  the  absence  of  noise: 


®  2.  (D 

@ 

©  »■  @ 

2 

2.  2.  3 

(D 

(D  s  ® 

This  scheme  satisfies  the  conditions  for  coding,  for  single-error 
detection.^  Thus  cl  Cm)=  z  ,  and  the  rate  of  computation  is 


'Note  that  choice  of  {l,  3}  in  this  case  preserves  separation  between 


code  points.  Thus  if  d(l,3)  =  distance  between  1  and  3,  then 
d[f(l,  3).f(1.3)]  =  [d(l,3)l. 
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R  =  b/s  where  N(M)  a 

number  of  elements  in  n,  and  »vw  ss  number  of  truth -values  in  the 

logical  scheme.  We  note  that  the  product  of  rate  and  minimiun  distance  is 
given  by  RdCM}  =  2 /.Cby^S  •  With  t(Vl=  5  .and  M  = 

we  obtain  the  following  mapping: 


& 

®  a.®4-  © 

0  2.  ®  4  (D 

2 

2  X  -S  4  5 

® 

®  3  ®  4-® 

4- 

4  4  4*  4  5 

(D 

®  S'  ©5® 

This  allows  us  to  detect  single  errors  at  a  rate  R  J 

bits/symbol,  di(M)=  2_  »  and  Rcl£M)”  .  Choosing 

M=  (t,Sj  allow.s  for  single-error  correction^  at  a  rate 

,  with  =  ,  hence  fid  Ct^  ^ - 

Similarly,  with  tH  *  9  >  M  =  £  1,^ ;  j  *  obtain  the  following 
scheme: 


3®  fir  ®  7  e® 


© 

2 

3 

S 

® 

7 

e 

® 


01  9®( 
2.  2  3  4-  5r 
3  3  3  4-  S' 
04  4®  S 
5  5  f  S  S 

(Db  b  0^ 

7  7  7  7  7 

esses 

9  ®9 


®  7  a® 
b  7  9  9 
b  7  «a 

?7  S® 
7  9  9 
0  7  S® 

7  7  69 
6  0  99 

®  9  9® 


^By  performing  maximum  likelihood  computations,  on  the  basis  of 
minimum  Hamming  distance  between  "received*  points  and  code  points. 
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which  permits  correction  of  some  single  errors,  and  detection  of  all 
others,  at  the  rate  R=  with 

RdiM)  «  a/  lo^3  .  This  compares  favorably  with  the  3-valued 
scheme,  which  permitted  single-error  detection  only  at  the  rate 
164  3  •  Alternatively,  the  9 'valued  scheme  permits  single¬ 

error  aetection  at  this  rate,  with  a  lower  probability  of  error  Pe 

For,  while  d  (m;  is  constant,  the  mean  distance  between  code  points 
in  has  clearly  increased.  With  a  27-valued  Post  disjunction,  and 
with  -  t  •  j  »  ^7  3  >  single  errors  can  be 

detected  at  the  rate  but  now  olCM^s  3  .  and  the  mean 

code  point  separation  has  further  increased,  consequently  Pe  has 
decreased. 

In  general,  for  the  M-valued  Post  disjunction,  the  mean  separation 
between  code  points  (that  is,  the  mean  nearest-neighbour  separation)  is 

.  Since  f?  ^  i  ,  in  the  noisy  case, 
sa  00  In  particular,  for  =  Rs  1  , 

SfM)=  -^S£M)asOo  .  Thus,  the  mean  sepa¬ 

ration  between  code  points  increases  with  yvt  (or  n.  ),  independently 
of  the  rate.  Since  the  probability  of  error  Pg,  associated  with  the  coding 

scheme  is  inversely  proportional  to  "SfMj,  this  result  would  seem  to 

imply  that  ^  approaches  zero,  independently  of  rate.  This  is  not, 
however,  the  case,  as  examination  of  Post  logic  clearly  demonstrates. 

The  structures  that  we  have  considered  are  all  mappings  from  direct 
products  of  chains,  onto  chains  (see  Appendix); 


I  I 


I 


I 


* 


1 


Figure  17 


Increasing  fiL  ,  in  such  a  way  that  this  chain  structure  is  preserved. 
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implies  a  transition  from  systems  with,  say,  3  states,  ordered  as  fol¬ 
lows: 


Figure  18 

to  systems  with  m.  states,  ordered  as  follows: 


Figure  19 
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In  the  limit,  we  approach  systems  with  a  continuum  of  states.  In 
terms  of  our  model  for  computation,  we  have  replaced  the  binary  scheme 
(Figure  IZ)  by  the  following  m-ary  scheme: 


Figure  20 


This  means  that  we  have  replaced  a  binary  channel  (BSC),  by  a  noisy 
ta-ary  channel: 


Figure  21 
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We  can  only  gain  by  this,  if  the  relative  capacity  of  this  channel  is  not 
less  than  the  relative  capacity  of  the  binary  channel.  This  will  be  the 
case  (roughly)  if  the  transition  probabilities  between  nearest  states  of 
the  m-ary  channel  are  of  the  same  order  as  those  of  the  binary  channel 
Thus  effectively,  the  m-ary  scheme  is 


Figure  22 


and  all  other  tran.sition  probabilities  are  small  enough  to  neglect. 

This  condition  is  realizable  with  m-ary  state  schemes,  if  the  noise 
fluctuations  between  neighbouring  states  are  independent  of  M.  In 
genersd,  this  is  not  true  as  M  increases  indefinitely,  since  components 
and  transmission  lines  are  energy-bounded,  and  we  are  limited  to  a 
fixed  range  in  which  to  place  states.  As  the  number  of  states  increases, 
the  fluctuations  between  states  become  larger,  and  so  the  m-ary  state 
channel  of  our  model  becomes  effectively  noisier.  We  can  compute  this 
effect,  roughly,  by  normalizing  Si  •  We  note  that  if  Ajis  the  dis¬ 
tance  between  states  in  the  3-valued  scheme  (so  that  the  total  range  is 
of  length  3A3),  then  the  distance  between  neighbouring  states  in  an 
m-ary  scheme  is  Am,  »  lAs  /"r)!— 'i 
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Figure  23 

Then  ^  (  M)  normalized  becomes 

Choosing  our  unit,  we  obtain  S*tM)  =•  (awi/w-i  J-1  /  I 

for  large  rfL ,  so  that  S'^Cm)  »  0  .  This  implies  that  if  we 

keep  the  rate  constant.  Pe  foes  tol ,  not  to  0  .  This  result  is 

obvious,  since  OO  implies  ansilog  computation,  and  the  notion 

of  code  redundancy  is  not  meaningful.  Evidently,  we  have  to  keep  Vrt, 
fixed,  and  small,  and  investigate  possibilities  of  obtaining  Post  functions 
with  requisite  minimum  distance  properties,  by  constructing  redundant 
nets  of  m»  state  components.  We  shall  consider,  first,  those  results 
obtained  by  the  use  of  redundant  parallel  nets  of  2.  state  components,  that 

1  2-4 

is,  nets  considered  by  von  Neumann  and  McCulloch  et  al.  It  has 

been  shown,^  that  the  redundant  automata  constructed  by  von  Neumann 
can  be  characterized  by  Lewis'  many-valued  logicsd.  schemes.  For 
example,  the  following  redundamt  ■and'*  network: 
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Figure  24 


can  be  characterized  by  the  4-valued  Lewis 


under  the  correspondence 

/  11  10  01  oo 
\  1  X  3  ^ 
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In  lattice  theoretic  ternns,  this  is  equivalent  to  the  mapping: 


Figure  25 

where  the  specificity  of  ^  is  characterized  by  the  following  map: 


Figure  26 
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Suppose  we  choose  M  — ^1^43  •  Then  we  obtain 


$ 

0  2  3  0 

6) 
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3  4  3  4 
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0  4  4  0 

which  satisfies  the  necessary  coding  requirements.  We  note  that  $  preserves 
minimum  distance  from  ^  "Xj.  to  8rX»  .  That  is,  the  dis¬ 
tance  between  subsets  in  ordered  under  is  equal  to 

the  minimum  distance  between  distinct  points  in  either  or  For 

tliis  scheme,  we  can  evidently  detect  single  errors  at  a  rate 
R  =  =  </£.  ,  where  iCM)  =2  . 

For  M  =  2^  (3  lines  per  bundle), 


Xl  iLi- 


Figure  27 


we  obtain  the  8 -valued  Lewis 
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which  is  equivalently  a  mapping  characterized  by 


Figure  E8 


under  the  correspondence: 


(Id  no 


(01  too  on  oio  ooi  ooo 

3  -4.  S'  b  7  8 


When  we  come  to  choose  our  message  set  M  ,  we  find  that  the  only 
set  that  satisfies  our  requirements  is  ‘ 
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We  can  thus  correct  single  errors  at  the  rate  R  s  JLoaZ / JImQ  a  (/3^ 

with  S  C  M  )  =  3  •  * 

Similarly,  for  tH  =  ,  we  obtain  the  16 -valued  Lewis  . 

The  only  M  -sets  available  are  M  ®  {  I  ^  (B  /  lb  3  *  ®  subset  of 

this,  H  '  »  1  •  ;lb3  .  The  former  has  ^  (  M  )  2.  ,  so  single  errors 

are  detectable  at  the  rate  R,  /bjlb  =  »/2.>  as  before.  The 

latter  has  }  s  4-  ,  so  we  can  correct  single  errors  and  detect 

double  errors  at  the  rate  R.  a  Aoj X  /'^b«  lb  ~  I  /4.  .  Alternatively, 

we  can  correct  single  errors  at  l/i*.  i  ,  with  lower  pL  , 

than  at  R  =  '  /3  .»  »TV  ■=  2^  • 

The  nature  of  this  result  is  clear.  The  coding  conditions  are  always 
violated  by  any  point  set  other  than  t  d  l*'j 

(n  even)  or  tda''3  to  odd  )  ,  for  the  following  reason: 

It  should  be  evident  that  the  trutn-value  lattice 
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8 


Figure  29 

is  itself  a  representation  of  the  8 -valued  Lewis  (and  its  dual  opera¬ 

tion,  the  Lewis  "or"  function,  ‘tjf*  ).  Thus  the  infinum  (g. Lb.)  of  any 
pair  of  lattice  points  is  their  ^  For  example,  3.  $  3 
287  =  8.  S%7  ^  7  ^  9  ,  and  so  on.  If  we  now  choose 

any  set  other  than  M  »•  ^  li  0^  >  say 

,  which  has  'iCM)  **•  then  closure  is  violated, 
since  =  9  This,  in  general,  will  sdw  ays 

occur  for  odd-jot  functions  of  2  variable  logic,  since  they  possess  the 
lattice  property  of  always  mapping  any  pair  of  points  into  a  point  an  odd 
number  of  units  of  distance  away.  Thus  the  only  point  sets  not  violating 
the  closure  condition  are  those  point  sets  containing  only  the  points  1| 
and  the  extremal  median  lattice  points  whose  intersection  is  the  point  3 
Thus  the  rate  for  error  correction  for  these  schemes  is  always,  at  most. 

For  the  even-jot  functions,  these  restrictions  do  not  apply.  Thus  if 

3 

we  consider  the  following  net;  For  m  =  2  (3  lines  per  bundle). 
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Figure  30 

we  obtain  the  8 -valued  Lewis  function 
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and  if  we  now  choose  M  r=.  ^  the  codability  conditions 

are  satisfied,  and  single  eirois  can  be  detected  at  the  rate 
(5.  ss.  £oj^/ Si  A  /3  ,  with  S  C  M  ^  =  2.  •  Similarly,  if  we 
form  the  16-valued  Lewis  function  (4  lines  per  bundle),  we  obtain  the 
function 


122 


0 

2 

5 

@ 

5 

(D 

0 

B 


14 


0a.s@5O@8  9@®«2(J|)i4 
X  I  4  3  b  &  e  7  10  9  a  II  14  13 


341  ^7  8  5b  1(12^10 
@320  9®©S  ll®®9 

5  b  7  0  I  2  3  4  IB  H-  IS  lb 

®5e®x0  03  l4([i)([C)lS 

•005‘®1»©0'a.  lS(g)  @14 

6  7  b  S  4  3  2  J  lb  15  14  13 

9  10  II  12  13  14  IS  ib  I  2  3  4 

Q  9  a©  14  @  ®  IS  2  0  0  3 

©a9®l5®0)l4  3©01 
12  11  lo  9  Ib  15  14  13  4-  3  2  I 

@14I5@^@([[)i2.  £•  ^0  g 

14  13  14  15  lO  ^  12  H  b  5  6  7 


15  Ib 

©  15 
9  10 
®  9 
®  a 

12  II 
5  b 

CD  5 
®  8 
©  7 

O 

2  1 


IB  ® 

Ib  15 
15  14 
14® 

11  12 

12  ® 

9  <S) 

10  9 

7  8 

b  & 

^  © 
4  3 


1 5  lb  (3  14  II  *2  9  lo  785-4  34  /  a. 

@  IS  14®  120®  9  9  @©  5  ®  3  2.  © 


If  we  now  choose  M—  C  *<41  by 7/  lOy  Ilyi3,  IbJ ,  single-error 
detection  is  possible  at  the  rate  ft  =:  |b  —  5/4  >  with 

SCM)  =  .  In  fact,  in  general,  single-error  detection  in  a  2*^  valued 

Lewis  logic  can  be  performed  for  even  jots  at  the  rate  ft  —  0^=l-Vn> 

Similarly,  single-error  correction  can  be  performed  at  a  rate  of,  at 

most,  fts:  |-  l-H^)  ,  and  so  on.  Alternatively,  the  rate  can 

be  kept  constant,  in  which  case  the  minimum  distance  increases,  and 
hence  Pg  decreased. 

In  terms  of  the  lattice  picture,  this  resxilt  is  possible  because  of  the 
fact  that  Lewis  functions  composed  of  even-jot  functions  are  such  that 
any  pair  of  points  maps  into  a  lattice  point  an  even  number  of  points 
away  (apart  from  the  set  which  is  always  closed  because 

Lewis  logic  is  functionally  incomplete).  The  particular  coding  used  by 
von  Neumann,  "bundling"  and  setting  a  fiduciary  level  so  that  R— 1/10*  , 
can  be  characterized  by  the  following  diagram: 
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Figure  31 


This  is  different  in  character  from  the  previous  schemes, in  that  there  is 
no  exclusion  of  possible  inputs.  The  set  ,  as  such,  does  not  exist. 
What  is  being  sought  is  a  net  which  maps  that  which  is  designated  as 
information  ,  into  itself,  and  that  which  is  designated  as  malfunc¬ 

tion  i  I  3  into  itself.  For  a  fuller  discussion  of  this,  see  Cowan. ^ 

'  2-4 

The  scheme  of  McCulloch  et  al.  uses  essentially  the  same 

principle  as  this,  but  has  a  different  structure.  A  more  efficient  error- 

reducing  scheme  results  from  the  higher  connectivity  of  the  structure 

(see  Cowan^). 

Both  schemes,  however,  are  of  the  same  character  as  the  previous 
one,  in  that  goes  to  zero  with  R  .  All  of  these  schemes  are 

characterized  by  a  certain  symmetry  —  that  the  connectivity  pattern  for 
each  neuron  is  identical  for  each  scheme.  We  shall  therefore  investigate 
an  asymmetric  system 
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Figure  32 

Let  each  neuron  fire  for  inputs  of  strength  2,  or  more.  Thus  we  have 

I  *ai  *21. 
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We  combine  these  by  taking  their  Kronecker  product: 
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where 

Xi  “  f  J  Xi,  =  ~  ia.3  ■ 

We  now  introduce  the  correspondences: 


oi  oo 
3  4 


and 


and  obtain  the  4 -valued  Post  function: 
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The  only  subsets  not  violating  condition  1  arc  {‘,ij  and 
neither  of  which  satisfies  condition  2.  . 

Similarly,  the  following  scheme  (a  modification  of  McCulloch's  nets) 


*ii  itij. 


Ha 


Figure  33 

realizes  the  following  Post  function: 
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This,  again,  is  an  unsatisfactory  mapping.  It  appears  that  any  scheme 
with  symmetric  weights  on  the  inputs,  and  noninteracting  bundles  will 
not  work.  However,  more  work  needs  to  be  done  on  this. 

We  now  consider  schemes  with  components  possessing  more  than 
two  stable  states;  and  we  consider,  first,  equal -input  schemes,  with 
M  =  3. 


X||  35*1 


Figure  34 
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Each  "neuron"  has  3-stable  states,  and  represents  a  Post 


Taking  the  Kronecker  product  of  this  v/ith  itself,  we  obtain 
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which  leads  to  the  9 -valued  Lukasiewicz -Post  function: 
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under  the  correspondence 
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Let  us  choose  M  =  ,  so  that  12  Si  £0^4./^^  9 

ic ;  R.  ss  1  /  £o«|  ^  3  Wc  obtain  the  following  scheme: 
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This  is  the  same  mapping  (as  far  as  informationally  significant  truth 
values  are  concerned)  as  M  »  $  i  *  1 .  The  rate  ftsl  is 

equal  to  that  for  w\  However,  the  set  ^ 

f  »  is  s  chain,  but  is  itself  a 

lattice.  That  is 


Figure  35 

Since  cL  (  h  ^  ^  ^  i  9  )  ^  ^  ,  condition  2.  is  violated,  so 

that  f  1/4.3  and  i  are  not  separate  information  points.  We 

might  just  as  well  smd  only  f  I  i9j  at  a  rate  1  /  a.  £0^2  3 

Let  us,  however,  choose  ^  =  th^,7,9l  .  Then  we  obtain 
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We  thus  satisfy  the  coding  conditions,  so  that  single-error  detection  is 
possible,  at  the  rate  ft  1  /  406^3  ,  with  dfM)  2.  :  thus 

2.  /  ..£05^3  .  Had  we  chosen  M  i  .  A  3  ' 

single-error  correction  would  be  possible  at  the  rate  ft  a  » 

with  so  that  ftdl  C  M  )  —  »  as  before.^ 

Similarly,  choosing  yn  a.  «  27  ,  and 


Figure  36 
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Single-error  detection  is  possiDie,  at  the  rate  Id*  I 
and  ftd  (M)  »  ,  as  before. 

If  we  compare  these  schemes  for  tW»3  »  so  that  H  2 /l^a.  3 

with  those  based  on  At*  Zi  wherein  J.  el.  ,  it 

would  seem  that  a  general  relationship  between  wi,  R  and  d  CM  )  exists. 
.In  fact,  for  ,  if  we  choose  M  ss.  t  j»3,/b3  ’ 

we  obtain  the  following  scheme; 


Figure  37 


Thus  ^  =•  •®09i,4'/^eyjlb  =  t/Zi  and  d^M)=3,  so  that 

dCM)  “  SxZ  a  3  /  general,  then,  we  can  induce 

that  H.<i(  M)  s,  -eo^x,i^  (There  are  other  reasons  for 

the  existence  of  this  relation,  based  on  notions  of  order  continuity,  and 
suchlike,  which  we  shall  not  discuss  here.)  We  have  not,  however, 
normalized  ctC  M  )  ,  to  account  for  limitations  on  the  range  available 

per  component.  If  we  perform  this  normalization,  then  ct  C  M  )  nor¬ 
malized  ie  given  by  M  )  ~  d  C  M  )  • 

f  1  4  T  *  m  — 1  i  .  The  parameter  -f  measures  the  effects  of  different 
ratios  of  ranges  for  rtv  states, to  that  of  the  H.  state  range.  Thus  ^  =  1 
implies  that  the  distance  betv/een  neighbouring  states  remains  constant 
and  the  range  is  unbounded,  while  vM-'l  implies  that  the  range  is 
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fixed.  These  two  limiting  cases  are  of  special  interest. 


Thus 
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1  V*  1 


m-l 


and  therefore, 


^  1 

y*-  a  =  CO 


we  can  plot  these  bounds  as  follows: 


Figure  38 


This  figure  can  be  interpreted  as  follows:  Let 

Then,  since  Pfi.  varies  inversely  with  (A  (  M  )  »  and  since 
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dCM)  i  .  But,  in  general,  for 

thes«  product  schemes,  ft  s  i  £o^j^w\j  ,  hence 

Rft  «v  /  sc  iM.)  When  *r»M-l.  . 

s:  '\  /£o^^w\/  >  and  3/v  »  which  is  independent  of  v»U  . 

Hence  we  gain  nothing  by  introducing  Yfl  statea  For  example, 


Figure  39 

When  -rai  ,  (tW-l  ) ,  and  Pg  2. /a  ^  , 

and  we  do  gain  something.  However,  from  the  previous  result,  cdl 

we  need  to  use  are  the  first  and  last  states,  so  that,  agaio.  nothing  is  to  be 
gained  by  using  the  W  state  components  in  multiplexed  nets.  For 
example, 
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Figure  40 


It  is  not  to  be  understood  that  many-valued  logical  schemes  are  not  useful, 
but  merely  that  parallel  nets  of  vrWstate  schemes  are  no  better  than  parallel 
nets  of  i,  state  components,  as  far  as  the  correction  of  all  errors  is  con¬ 
cerned. 


SUMMARY  AND  CONCLUSIONS 


It  is  clear  that  although  a  nonzero  computational  capacity  (or  rate, 
for  that  matter)  exists  in  principle,  its  practical  realization  is  another 
matter.  For  probabilistic  logical  functions,  use  of  the  model  consisting 
of  a  series  combination  of  an  error-free  lo^cal  function  and  a  noisy 
binary  symmetric  channel  leads  to  the  conclusion  that  the  realization  of 
nonzero  rates  requires  the  matching  of  the  output  of  the  error-free  com¬ 
putation  device  to  the  noisy  channel.  Assuming  the  existence  of  error- 
free  encoders  and  decoders  (which  itself  is  another  problem),  and 
performing  only  coordinate  encoding  (to  ensure  that  no  "cheating*  occurs 
by  way  of  computation  in  the  encoder),  we  investigated  various  many¬ 
valued  logicsd  schemes,  none  of  which  realized  a  nonzero  computation 
rate  for  arbitrarily  reliable  computation. 

However,  we  neglected  any  considerations  of  source  statistics  and 
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considered  only  stnactures  formed  from  parallel  nets  of  identical  neurons, 
thus  duplicating  the  previous  results  obtEdned  by  Ellas.  We  then  con¬ 
sidered,  in  a  preliminary  fashion,  slightly  more  complex  nets,  in  which 
each  neuron  computed  a  different  function,  but  were  not  able  to  utilize 
these  schemes.  Finally,  we  considered  parallel  nets  of  M-state  com¬ 
ponents,  and  found  that  nothing  was  gained  by  using  the  (M-2)  intermediate 
states  between  1  and  M,  which  leads  to  these  same  negative  resxilts. 

The  conclusions  to  be  drawn  from  this  preliminary  analysis,  are 
not  that  a  nonzero  rate  is  nonrealizable,  but  that  the  schemes  considered 
here  are  too  simple-minded,  and  it  may  be  possible,  for  example,  that 
nonzero  rates  can  be  obtained  from  nets  consisting  of  M-state  components, 
with  each  neuron  computing  a  different  function,  the  whole  operating  on 
input  "bundles"  that  are  not  simply  noninteracting,  but  weight  input  proba¬ 
bilities  in  some  fasMon.  To  this  end.  the  following  scheme  is  being 
investigated: 


Xi  X,. 


Figure  41 


137 


APPENDIX 


MANY-VALUED  LOGICS 


Many-valued  logics  (or  non-Aristotelian  logics)  were  first  con- 

17  18 

structed  by  Post,  and  independently  by  Lukasiewicz.  Post's  scheme 

was  a  straightforward  generalization  of  Boolian  logic.  Compared  with 

Boolian  functions,  which  take  values  in  the  set  {l,  O},  or  equivalently 

{l,  2},  Post  functions  take  values  in  the  set  {l,  •  •  •  »  ^ »  O}  or 

equivalently  {l,  2,  3, . . . ,  m-1,  m}.  At  most,  there  are  m  Post  func- 

2" 

tions  in  n  variables,  compared  with  the  2  possible  n-variable  Boolian 
functions.  Many  Post  functions  are  natural  generalizations  of  Boolian 
functions.  For  example,  the  Boolian  disjunction  and  conjunction  can 
be  represented  in  truth-value  matrix  form  by 


respectively,  while  the  corresponding  Post  functions  are  represented  by 
the  matrices 


Both  Boolian  and  Post  disjunction  and  conjunction  can  be  represented  by 
the  tinath-value  functions 
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=  iif  Cx,f) 

df 

Similarly,  Boolian  and  Post  negation  caui  be  represented  by  the  troith- 
value  function  X  We  note,  however,  that  in  an 

M-valued  Post  logic,  there  are  xn™  unitary  functions,  compared  with 
2 

2  unitary  Boolian  functions.  The  most  useful  of  these  Post  functions  is 
the  set  defined  as 

acn  s  I 


There  also  exists  an  analog  of  the  fundamental  theorem  of  Boolian  logic, 
which  fitates  that  every  Post  function  can  be  expressed  as  the  conjunction 
of  disjunctions  of  J  functions.  That  is, 

VM  M 

in»1 

m" 

Since  there  are  m  Post  functions  of  n-variables,  this  implies  that 
every  m  X  m  truth-value  matrix  is  a  Post  function  of  two  variables,  and 
so  on.  For  this  reason,  Post  logic  is  said  to  be  functionally  complete. 

The  many-valued  logics  of  Lukasiewicz  have  the  same  disjunctions, 
conjunctions  and  J-functions  as  Post  logic,  but  are  functionally  incom¬ 
plete.  That  is,  not  every  Post  function  is  a  Lukasiewicz  function. 
Lukasiewicz  functions,  in  fact,  sati.3fy  the  constraint  that 
The  truth-value  set  e  {  1/  ,  is  closed,  under  any  composi¬ 

tion  law  corresponding  to  some  Lukasiewicz  function.  Because  of  this 

constraint,  there  are  only  ^  “  possible  Lukasiewicz  functions 

of  n-variables.  A  further  form  of  many-vsilued  logic  was  constructed  by 


139 


19 

Lewis.  This  logic  is  essentially  a  Boolian  product  logic,  whose  struc- 

'  cj  ?o 

ture  is  isomorphic  to  that  of  a  Boolian  eilgebra  of  order  2.  • "  That  is, 

a  22,^  valued  Lewis  logic  in  rVvariahles,  has  a  structure  isomorphic  to 
that  of  a  JL  valued  Boolian  logic  in  variables.  There  are  thus 

possible  WW  valued  Lewis  functions  in  tV  variables. 
Lewis  logic  is  functionally  incomplete  and,  in  addition,  possesses  other 
symmetry  conditions  because  of  its  structure.  Thus  for 

and  for  ms9fj  t  C  4-)  i  k  )  17  ) 

The  nature  of  these  symmetry  relations  will  be  made  clear  in  the  following 
discussion. 

Examples  of  Lewis  functions  for  KM  =  4*  are  given  by  the  Lewis 
disjunction  and  conjunction: 


The  differences  between  these  functions  and  the  corresponding  Post- 
Lukasiewicz  functions  are  marked. 

Important  unitary  functions  are  the  Lewis  J  functions  defined  by  the 
following  table: 


X 

J3 

J4 

1 

1 

+ 

4 

4 

a 

4 

2. 

J 

4 

3 

4 

3 

2. 

4 

4- 

4 

4 

4 

'I 

and  also  the  Lewis  model  function  "Possibly  *5C  “  COxj 
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(c : 


I  4  t 


The  fundamental  theorem  for  Lewis  logic  is  similar  to  that  for  Post  logic, 
with  additional  constraints  on  f  MjU,— Alternatively,  any 
Lewis  function  can  be  expressed  by  some  combination  of  the  triple 
^  V  j  ^  ^  more  extensive  discussion  of  these  logics  can  be 

found  in  Lewis^  and  Kiss. 

A  lattice  theoretic  characterization  of  many-valued  logics  can  be 
20  21 

made  (see  Kiss  and  Birkhoff  ).  All  the  logics  considered,  thus  far, 
can  be  represented  as  complemented,  distributive,  modular  lattices. 

Thus  the  Zy  unitary  functions  of  3 -valued  Post  logic  can  be  charac¬ 
terized  by  the  following  lattice: 


Figure  42 
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The  12-unitary  functions  of  Lukasiewicz  logic  are  also  shown  in  this 
lattice: 


Figure  43 


Clearly,  this  lattice  is  also  a  sublattice  of  the  Post  lattice. 

The  lattice  of  valued  unitary  functions,  ib  in  all,  can  be 
represented  as  follows: 
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Figure  44 


which  is  also  a  representation  of  the  Boolian  lattice  of  2-variables.  This 
lattice  is  a  sublattice  of  the  lattice  of  64  Lukasiewicz  4-valued  unitary- 
functions,  which  is  itself  a  sublattice  of  the  256-elemeni  Post  lattice  of 
4-valued  unitary  functions. 

Another  lattice  characterization  of  these  many-valued  logics  (which  is 
more  useful  for  our  purposes)  can  be  given.  Functions  are  now  regarded 

as  mappings  from  a  lattice  of  variables  into  a  lattice  of  truth -values.^ 
Thus  our  unitary  functions  are  represented  by  the  following  schemes; 
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4 


1 


4  4 


Post -Lukasiewicz  functions  Lewis  functions 

Figure  45 

The  extra  symmetry  conditions  possessed  by  Lewis  logic  are  now 
obvious.  The  truth-value  lattice  in  Lewis  logic,  is  nondegenerate,  and 

22 

forms  a  partly  ordered  set.  The  lattices  of  truth  values  in  Post- 
Lukasiewicz  logic  are  all  degenerate  chains,  and  lack  these  symmetry 
condition  fi. 

Similarly,  the  many-valued  binary  functions  can  be  represented  as 
follows: 


11 


3 -valued  Post -Lukasiewicz  functions 
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11 


44 


4 -valued  Lewis  functions 
Figure  46 

Finally,  although  the  Post-Lukasiewicz  schemes  of  truth-values  grow 
in  monotonic  fashion  with  m 


F'lgure  47 
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the  Lewis  schemes  grow  as  follows; 
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APPENDIX 


Ihe  following  questions  were  answered  during  the  discussion  period 
after  this  presentation  and  are  included  here  to. more  fully  explain  or  aug¬ 
ment  portions  of  the  paper- 

Questions  "Pleiase  expand  upon  the  relation  between  dimension  and  entropy  of 
a  space.  Is  it  not  possible  to  have  two  different  dimensional 
spaces  with  the  same  entropy?"  (Dr.  L.  Fogel,  National  Science 
Foundation. ) 

Mr.  Cowan:  I  have  not  really  made  any  use  of  the  concept  of  dimension,  but 
merely  considered  (intuitively)  the  number  of  Independent  variables  in  the 
X  and  Z  spaces  to  be  a  measure  of  their  dimension.  That  is  the  logon  content 
of  X  and  Z  is  a  dimensional  measure.*  If  we  now  assume  that  the  inetron  con¬ 
tent  per  logon*  is  a  constant,  it  follows  that  H(X)>H(Z)  if  and  only  if 
dim  X>dim  Z.  There  is  another  sense  in  which  dimension  and  entropy  are 
related,  but  this  is  concerned  with  distributions  which  may  be  partly  contin¬ 
uous  and  partly  discrete,  and  I  refer  you  to  Renyi^^.  The  answer  to  the 
second  part  of  your  question  is  yes,  but  under  circumstances  which  are  not 
relevant  to  the  considerations  of  this  paper. 

Question:  "Gan  you  comment  on  the  possible  relation  of  your  last  hypothesis 
to  Taylor's  analog  scheme?"  ** 

fir.  Cowan:  There  is  no  immediately  obvious  relation  between  these  schemes. 

It  must  be  emphasized  that  we  have  been  concerned  only  vdth  some  mathematical 
aspects  of  certain  highly  formalized  automata.  It  is  true  that  the  components 
of  these  automata  are  formal  models  of  nerve  cells,  but  they  are  very  incom¬ 
plete  models.  Only  those  properties  that  can  be  properly  handled  in  a  Boolian 
calculus  are  utilized.  The  ultimate  aim  of  this  work  is  to  discover  those 
principles  of  organization  which  produce  reliability  of  function  in  the  pres¬ 
ence  of  noise.  We  are  therefore  not  too  interested  in  automata  which  perform 
complex  fiuictions  (for  the  present),  but  are  concerned  only  with  those  automata 
that  perform  th?  simplest  functions.  Dr.  Taylor's  scheme,  it  seems  to  me,  is 
an  attempt  to  discover  those  principles  of  organization  possessed  by  automata 
which  perform  such  complex  functions  as  pettern  recognition  and  learning. 
>breover  the  method  of  attack  is  very  different.  The  known  properties  of 
neinre  cells  are  embodied  in  electronic  models,  and  automata  are  synthesized 
from  such  models.  The  resulting  automata  appear  to  be  able  to  "recognize" 
patterns,  rnd  to  "learn,"  but  it  is  not  knovm  hov  these  functions  are  performed. 
In  the  schemes  we  have  presented,  the  mechanism  whereby  reliable  performance  is 
obtained  is  quite  obvious. 


^  cf.  D.  M.  Mackay,  "Quantal  Aspects  of  Scientific  Information"  1st.  London 
Symposium  on  Information  Theory,  London,  1950. 

cf .  V;.  K.  Taylor,  "The  Simula ti on  of  some  Nervous  .^stem  Functioning" 

3rd.  London  Syiriposium  on  In^'orm.oUon  Theory,  London,  1956. 
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Question:  "In  your  discussion  you  limited  your  input  to  radix  2  and  then 
converted  to  radix  m.  What  if  the  primary  message  is  m?" 

(Dr.  E.  E.  Loebner,  Radio  Corporation  of  America) 


Answer;  I  am  sorry  that  this  point  was  not  clearly  made.  Ihe  schemes  con¬ 
sidered  were  in  fact  m-valued  functions  of  m-value  variables.  For  example, 
the  Lewis  functor: 


was  utilized.  Ibis  was  realized  ly  the  following  formal  neural  net: 


\diere  each  of  the  lines  and  components  of  the  net  can  support  two  states, 
the  net  is  so  organized  that  it  realizes  the  given  4-VBlued  function. 


But 


An  m-valued  function  of  2-V8lued  variables  ia  also  easily  obtained. 
Thus  the  follo:-dng  scheme 
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realizes  the  functor 


»1 


i 

T 

o 


1  o 


1  *. 
3  ^ 


*1. 


under  the  usual  correspondence.  We  have  in  fact  been  investigating  the 
utility  af  such  schemes. 

Question!  "The  point  is  really  about  your  last  scheme,  vdth  the  random 
connection.  I  would  like  to  hear  more  about  what  advantage 
or  virtue  the  randomness  has  over  some  equidistant  spacing  of 
the  possible  connections."  (Dr.  S.  Pepert,  National  Hiysical 
Laboratory) 

Mr.  Cowan:  Let  me  put  it  this  way.  It  should  be  clear  that  the  aim  of  this 
work,  namely  the  construction  of  reliable  automata  from  unreliable  components, 
is  really  a  part  of  the  general  problem  of  obtaining  reliability  of  function 
(whether  it  be  computation  or  transmission)  in  the  presence  of  a  variety  of 
disturbances.  This  in  turn  is  within  the  domain  of  Information  Iheory  and 
of  Statistical  Mechanics.  Two  features  of  this  problem  are  immediately 
apparent.  Firstly  we  would  like  to  construct  nets  that  are  as  rugged  as 
possible;  i.e.,  nets  that  produce  minimum  distortion  of  function  given  max¬ 
imum  distortion  of  individual  components,  and  of  the  structure  of  the  net. 
Clearly  If  we  can  constiaict  nets  whose  function  is  little  perturbed  by  small 
variations  of  structure,  then  this  is  desirable.  Hence  we  do  not  insist  on 
precise  connectivity  in  our  nets,  but  permit  (local)  variations  of  connec¬ 
tion:  and  our  nets  don't  function  because  of  this  variability,  but  in  spite 
of  it.  Secondly,  we  have  to  live  with  elemental  chaos,  as  it  were,  and  to 
insist  on  such  precise  connectivity  is  to  insist  on  a  degree  of  order  which 
is  scarcely  credible,  in  those  automata  which  we  are  trying  to  model,  iind 
in  fact  smacks  of  physical  demonology.  It  is  thus  of  little  use  to  postu¬ 
late  the  existence  of  ideal  noiseless  components  combined  into  a  precisely 
connected  net.  We  have  to  put  the  chaos  in  at  the  beginning,  and  start  from 
there.  Furthermore,  if  we  consider  the  specification  of  such  model  neiiral 
nets  as  we  have  considered,  from  a  genetical  standpoint,  the  selective  in¬ 
formation  content  of  such  precise  schemes  is  incredibly  high.  By  allowing 
imprecise  connectivity  (within  certain  limits),  and  imprecisely  functioning 
components,  the  selective  information  content  of  such  schemes  is  consider¬ 
ably  lowered.  Finally,  while  we  have  stressed  the  point  that  we  are  only 
interested  in  nets  which  perform  siaple  logical  functions,  we  would  really 
like  to  use  them  (eventually)  to  investigate  more  complex  and  interesting 
phenomena  such  as  learnings  and  for  this  purpose  we  would  like  to  have, 
initially  at  least,  an  imprecise  connectivity  in  our  model  nets.  We  can 
do  no  better  to  summarize  these  points,  than  to  quote  from  a  famous  paper 
■py  Dr.  McCxUloch  and  W,  Pitts,*  viz:  "To  demdnstrate  the  existential  con¬ 
sequences  of  known  characters  of  neurons,  any  theoretically  conceivable  net 


*W.  Pitts  end  W.  S.  l-fcCullooh:  "How  we  know  Uhiversals:  Ihe  Perception  of 
Auditory  and  Visual  Forms"  Bull.  Math.  Biophysics,  Vol.  9,  1947. 
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embodying  the  possibility  will  serve.  It  is  equally  legitimate  to  have 
every  net  accompanied  by  anatomical  directions  as  to  viiere  to  record  the 
action  of  its  supposed  components,  for  experiment  will  serve  to  eliminate 
those  which  do  not  fit  the  facts.  But  it  is  wise  to  construct  even  these 
nets  so  that  their  principle  function  is  little  perturbed  by  small  pertur¬ 
bations  in  excitation,  threshold,  or  detail  of  connection  within  the  same 
neighborhood.  Genes  can  only  deterndne  statistical  order,  and  original 
chaos  must  reign  over  nets  that  learn,  for  learning  builds  new  order  ac¬ 
cording  to  a  law  of  use." 


